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Relationship  of  Bar  Examinations  to 
Performance  Tests  of  Lawyering  Skills 


Stephen  P.  Klein 
The  Rand  Corporation 


The  relationship  between  scores  on  a  typical  bar  exam  and  the 
ability  to  practice  law  was  investigated  with  485  applicants  who 
took  the  bar  exam  and  a  two  day  performance  test  involving  basic 
oral  and  written  legal  tasks.  Results  indicated  the  two  types  of 
measures  assessed  similar  but  not  identical  skills. 
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Relationship  of  Bar  Examinations  to 
Performance  Tests  of  Lawyering  Skills 

STEPHEN  P.  KLEIN 
The  Rand  Corporation 


BACKGROUND 


Most  bar  examinations  consist  of  a  nationally  administered  200  item 
multiple  choice  test,  called  the  Multistate  Bar  Examination  (MBE),  and 
several  state  developed  essay  questions.  Both  the  MBE  and  essay  assess  an 
applicant  to  the  bar's  knowledge  of  the  law  and  the  ability  to  analyze 
legal  problems.  The  essay  further  measures  an  applicant's  ability  to 
identify  legal  issues  and  express  ideas  in  a  "lawyer  like"  fashion. 

State  bar  examinations  have  been  criticized  for  measuring  only  a  few 
of  the  important  skills  and  abilities  that  are  needed  for  the  practice  of 
law.  For  instance,  a  typical  bar  exam  question  provides  all  the  facts  that 
are  auiterial  to  a  case  and  then  asks  the  applicant  to  determine  how  the 
case  should  be  resolved.  The  exam  does  not  assess  whether  the  applicant 
knew  what  questions  to  ask  in  order  to  obtain  these  facts.  This  is 
comparable  to  measuring  a  physician's  diagnostic  ability  on  the  basis  of 
his/her  skill  in  interpreting  test  results  without  ever  assessing  the 
physician's  judgment  in  knowing  which  diagnostic  tests  to  order.  Some  of 
the  other  skills  that  are  important  for  legal  practice  that  are  not  tested 
directly  by  the  traditional  bar  exam  include  the  ability  to  interview 
clients,  examine  a  witness,  conduct  legal  research,  and  negotiate  a 
settlement. 


State  bar  examinations  also  have  been  criticised  for  their 
disproportionate  negative  impact  on  ninority  groups.  In  California,  for 
instance,  the  percentage  of  Anglos  that  pass  is  about  twice  as  great  as  the 


The  views  expressed  in  this  paper  are  the  author's  and  do  not 
necessarily  represent  those  of  The  Rand  Corporation  or  the  agencies  that 
supported  this  research  (The  National  Conference  of  Bar  Examiners  and  the 
Ooaaiittee  of  Bar  Examiners  of  the  State  Bar  of  California). 


.*  v  .  Ja 


-  2 


percentage  of  Blacks  that  pass.  Although  the  differences  in  passing  rates 
among  groups  parallels  their  relative  performance  levels  in  law  school,  it 
has  been  argued  that  these  differences  would  be  substantially  reduced  if 
the  exam  measured  practical  (as  distinct  from  academic)  legal  skills. 

PURPOSE 

The  foregoing  considerations  led  to  an  investigation  of  the  following 

issues : 

o  Is  it  feasible  to  construct  psychometrically  sound  and  valid 
measures  of  clinical  (practice  oriented)  legal  skills  that  can  be 
administered  under  standardized  testing  conditions? 


o  To  what  degree  are  essay  and  MBE  scores  correlated  with  scores  on 
clinical  skills  measures? 

o  Are  scores  on  clinical  skills  tests  related  to  an  applicant's  legal 
training  and  experience? 

o  Would  the  use  of  clinical  skills  tests  on  a  bar  examination  narrow  the 
difference  in  passing  rates  among  racial/ethnic  groups? 


PROCEDURES 

The  foregoing  questions  were  investigated  by  having  applicants  to  the 
bar  take  a  battery  of  tests  administered  via  an  Assessment  Center  (AC). 

The  major  features  of  the  AC  were:  (1)  it  operated  for  two  weeks  beginning 
shortly  after  the  regular  bar  examination;  (2)  each  enrol lee  participated 
for  two  days  at  one  of  two  sites;  (3)  six  different  sets  of  tasks  (called 
problems)  ware  used  to  ensure  test  security;  (4)  in  three  problems, 
participants  functioned  as  the  attorney  for  the  plaintiff  in  a  simulated 
earns  while  in  the  other  three  problems,  participants  functioned  as  the 

for  the  defendant  on  a  different  case;  (5)  each  participant  took 
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one  plaintiff  and  one  defendant  problem;  (6)  all  three  problems  of  a  given 
type  were  based  on  the  same  detailed  test  plan;  and  (7)  each  problem  took 
about  6  hours  to  complete  and  contained  several  written  and  oral  tasks. 

The  tasks  were  developed,  pilot  tested,  and  revised  by  a  team  composed 
of  practicing  attorneys,  clinical  law  professors,  and  a  psychometrican. 
Specially  trained  actors  played  the  roles  of  clients  and  witnesses  for  the 
oral  tasks.  Oral  task  performances  were  videotaped  for  later  evaluation. 

A  s.trong  desire  to  participate  and  do  well  in  the  AC  was  achieved  by 
advising  applicants  that  a  high  score  in  the  AC  would  substantially  improve 
their  chances  of  passing  the  regular  bar  examination.  Over  4,000  of  the 
7,439  applicants  taking  the  July  1980  bar  exam  in  California  requested  to 
participate  in  the  AC.  A  sample  of  500  applicants  was  selected  from  this 
group  on  the  basis  of  several  factors  including  racial/ethnic  background, 
sex,  law  school  type,  and  amount  of  prior  clinical  experience. 


RESULTS 

Analyses  of  the  data  on  the  485  participants  completing  the  AC 
indicated: 

o  A  participant's  score  was  not  affected  by:  the  particular 
combination  of  problems  taken;  the  order  in  which  they  were 
taken;  warm-up  effects;  test  site;  possible  variations  in  how  the 
actors  played  their  roles;  or  whether  the  problems  were  taken  at 
the  beginning,  middle,  or  end  of  the  two  week  testing  program. 


o  The  pattern  of  correlations  among  tasks  on  one  problem  were 
almost  identical  to  the  pattern  of  correlations  among  tasks  on 
another  problem  of  the  same  type.  This  finding  and  others 
indicated  that  the  test  (and  scoring  guide)  development  teams 
succeeded  in  constructing  parallel  forms  for  each  problem  type. 
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o  The  attorneys  that  graded  the  AC  tasks  generally  agreed  with  one 
another  in  the  scores  that  should  be  assigned  to  an  applicant  (as 
indicated  by  an  analysis  of  variance).  Interevaluator  agreement 
was  higher  on  the  written  tasks  than  on  the  oral  tasks. 

o  The  written  tasks  tended  to  load  on  one  factor  and  the  oral  tasks 
on  another  (orthogonal)  factor.  The  average  correlation  among  the 
written  tasks  (r  =  .23)  was  higher  than  the  average  correlation 
among  the  oral  tasks  (r  =  .16).  Problem  type  did  not  emerge  as 
a  factor  (there  was  a  .93  correlation  between  problem  types  after 
correcting  for  attenuation) . 

o  Applicants  with  prior  clinical  experience  had  statistically 
significantly  higher  AC  scores  than  applicants  without  such 
experience  even  after  controlling  for  differences  in  mean  bar 
exam  scores  among  experience  groups.  The  pattern  of  correlations 
between  types  of  prior  experience  and  types  of  AC  scores  further 
supported  the  validity  of  the  AC  tests. 

o  AC  oral  and  written  scores  were  correlated  with  each  other.  They 
also  were  correlated  with  scores  on  the  essay  and  MBE  portions  of 
the  regular  bar  examination.  After  correcting  for  attenuation, 
the  correlation  between  the  oral  and  written  total  scores  (r  = 

.73)  was  higher  than  it  was  between  either  of  these  scores  and 
bar  exam  essay  or  MBE  scores. 


o  The  differences  in  average  score  among  racial/ethnic  groups  on 
both  parts  of  the  regular  bar  exam  generally  paralleled  the 
differences  among  these  groups  on  the  AC.  This  was  especially 
true  on  the  AC's  written  tasks.  Racial/ethnic  group  accounted 
for  only  a  very  small  percentage  of  the  systematic  variance  in 
either  AC  or  bar  scores . 

o  The  inclusion  of  AC  type  tasks  on  the  bar  exam  would  make  a  small 
but  noticeable  change  in  who  passed  versus  failed;  i.e.,  adding 
AC  type  tasks  would  result  in  some  applicants  going  from  a  pass 
to  a  fail  status  while  the  reverse  would  be  true  for  other 
applicants . 

o  Responses  to  a  post  AC  questionnaire  indicated  that  participants 
felt  the  AC  was  a  far  better  test  of  their  ability  to  practice 
law  than  was  either  section  of  the  regular  bar  examination. 

DISCUSSION  AND  IMPLICATIONS 

The  foregoing  findings  indicate  that  parallel  forms  of  psychometrically 
sound  performance  tests  of  important  lawyering  skills  can  be  constructed, 
administered  under  standardized  and  secure  test  conditions,  and  scored 
reliably.  The  findings  also  suggest  that  the  skills  and  abilities  measured 
by  the  AC  were  similar  but  not  identical  to  those  measured  by  the  regular 
bar  exam.  After  correcting  for  attenuation,  the  correlation  between  the 
AC  and  regular  bar  exam  (r  *  .72)  was  about  the  same  as  it  was  between  the 
essay  and  MBE  (r  *  .78).  Thus,  adding  AC  type  tasks  to  a  bar  exam  would 
result  in  a  small  but  systematic  change  in  who  passed  versus  failed. 


The  amount  of  change  would,  of  course,  depend  upon  how  much  weight  was 
placed  on  AC  tasks  in  making  pass/fail  decisions.  However,  even  if  a  great 
deal  of  weight  were  given  to  the  AC  type  tasks,  it  would  not  significantly 


increase  or  decrease  the  large  difference  in  passing  rates  that  currently 
exists  among  racial/ethnic  groups  on  the  bar  exam. 

Adding  a  practical  skills  component  to  the  bar  exam  probably  would 
substantially  increase  the  number  of  law  school  students  taking  courses 
that  were  specifically  directed  toward  developing  their  clinical  legal 
skills.  Adding  AC  tasks  also  might  improve  attitudes  toward  the  efficacy 
of  the  exam  on  the  part  of  the  legal  profession,  the  public,  and  the 
applicants  that  take  it.  Several  states  and  the  National  Conference  of  Bar 
Examiners  are  therefore  now  considering  changing  their  bar  examinations. 
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